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Abstract 


Q  methodology  is  a  research  method  with  a  proven  history  for  illuminating  agreement  and 
differences  among  individual  and  group  perceptions.  This  technical  note  describes  ways  for 
applying  Q  methodology  to  assist  software  engineering  processes.  As  a  project  management 
tool,  it  can  help  to  articulate  system  requirements  and  project  risk.  It  can  also  be  used  to 
identify  many  of  the  hidden  risks  and  costs  associated  with  system  efforts.  Q  methodology 
complements  a  project  manager’s  suite  of  methodologies  and  tools  by  providing  a  means  for 
uncovering  stakeholder  perceptions  of  incorrectly  specified  requirements,  looming  risks,  and 
hidden  costs.  In  doing  so,  it  provides  stakeholders  and  project  managers  with  additional 
insights  for  troubleshooting  project  threats. 
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1  Introduction 


1.1  Background 

Q  methodology  is  a  research  technique  that  allows  a  researcher  to  1)  identify,  understand,  and 
categorize  individual  perceptions  and  opinions,  and  2)  cluster  groups  based  on  their 
perceptions  [McKeown  &  Thomas  88], 

Invented  in  1935  by  British  physicist-psychologist  William  Stephenson,  Q  methodology  is 
most  often  associated  with  quantitative  analysis  because  of  its  reliance  on  factor  analysis 
[Stephenson  53],  Statistical  procedures  aside,  Stephenson  was  interested  in  providing  a  way 
to  reveal  the  subjectivity  involved  in  any  situation;  e.g.,  in  perceptions  of  risk,  appraisal  of 
costs,  perspectives  on  user  requirements,  and  opinions  on  training.  Q  methodology  allows 
researchers  to  examine  the  subjective  perceptions  of  individuals  on  any  number  of  topics.  It 
also  helps  to  identify  commonalities  and  differences  in  subjective  perceptions  across  a 
sample  group. 

The  real  utility  of  Q  methodology  lies  in  uncovering  these  opinion/perception  clusters.  Once 
identified,  they  can  be  targeted  for  follow-up  activities  such  as  further  research  or 
programmatic  activities.  It  is  the  combination  of  qualitative  and  quantitative  research 
techniques  that  allows  researchers  to  identify  individuals  who  share  common  opinions.  Q 
methodology  is  often  used  for  the  following: 

•  identifying  important  internal  and  external  constituencies 

•  defining  participant  viewpoints  and  perceptions 

•  providing  sharper  insight  into  preferred  management  directions 

•  identifying  criteria  that  are  important  to  clusters  of  individuals 

•  examining  areas  of  friction,  consensus,  and  conflict 

•  isolating  gaps  in  shared  understanding  [Steelman  &  Maguire  04] 

The  qualitative  aspect  of  Q  methodology  is  grounded  in  its  ability  to  emphasize  the  how  and 
why  people  think  the  way  they  do.  The  primary  goal  is  to  uncover  different  patterns  of 
thought — not  to  count  how  many  people  think  the  way  they  do  [Valenta  &  Wigger  97].  The 
quantitative  aspect  involves  using  factor  analytic  techniques  (specifically,  principle 
components  analysis  [PCA])  as  a  means  for  grouping  like-minded  individuals. 
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In  short,  Q  methodology  provides  analysts  with  “a  systematic  and  rigorously  quantitative 
means  for  examining  human  subjectivity”  [McKeown  &  Thomas  88]. 

In  terms  of  systems  development,  Q  methodology  may  be  useful  for  the  following: 

•  understanding  and  mitigating  pockets  of  resistance  in  system  adoption 

•  targeting  and  tailoring  system  features,  training  needs,  or  security  requirements 

•  isolating  data  standards  requirements  for  system  integration 

•  tailoring  system  performance  measures  and  metrics 

•  understanding  system  risk  elements 

•  tailoring  checklists  and  criteria  for  understanding  cost,  schedule,  and  sizing  estimates 

Section  2  provides  a  brief  introduction  to  Q  methodology  and  Section  3  illustrates  how  it  can 
be  used  in  a  software  context. 
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2  A  Primer  on  Q  Methodology 


Q  methodology,  like  many  research  methods,  can  be  used  to  observe  perceptions  from  the 
context  of  an  individual  or  from  the  context  of  a  group  of  individuals.  In  Q  methodology, 
intrasubjective  studies  gather  data  from  an  individual  on  multiple  issues  of  interest.  The 
individual’s  opinions  are  then  clustered  based  on  similarity  of  opinion.  The  purpose  is  to 
determine  whether  the  various  opinions  of  the  individual  give  rise  to  a  greater  thematic 
understanding  of  the  issues  at  hand.  Typically,  an  individual  may  be  asked  to  reveal  his/her 
perceptions  on  a  variety  of  different  constructs.  When  examined  in  total,  the  findings  may 
reveal  similarity  patterns.  For  example,  a  programmer  may  be  studied  to  determine  his/her 
preferences  for  different  software  programming  methodologies  under  various  conditions.  In 
this  case,  the  study  is  intrasubjective  because  the  researcher  is  studying  a  single  individual  to 
determine  if  preferences  cluster  around  one  or  more  common  themes. 

Conversely,  intersubjective  studies  focus  on  how  perceptions  of  groups  of  people  cluster  on 
one  issue  or  more.  The  issue  may  be  single  or  multidimensional.  The  point  being,  if  one  was 
only  concerned  about  how  various  traits  clustered  together,  they  would  be  employing 
traditional  factor  analysis  methods.  Alternatively,  Q  methodology  is  concerned  with 
clustering  like-minded  perceptions.  It  is  a  means  for  identifying  the  presence  of  patterns  of 
opinions.  Whether  it  is  single  dimensional  or  multidimensional,  the  study  is  always  framed 
around  finding  patterns  of  subjective  perceptions. 

As  illustrated  below,  the  methodology  features  three  distinct  stages  of  activities:  establishing 
the  Q-Sample,  administering  the  Q-Sort,  and  factor  analyzing  the  Q-Sorts. 


2.1  Stage  1 :  Establishing  the  Q-Sample 

Establishing  the  Q-Sample  involves  identifying  the  survey  items — not  the  participants  or 
statements  that  will  be  used  to  identify  individual  patterns  of  perceptions.  Because  the  survey 
items  are  perceptual  in  nature,  they  tend  to  reflect  subjective  statements  pertaining  to  the 
particular  research  area  of  interest.  (Later,  during  the  Q-Sort  procedure,  the  participants  will 
be  asked  to  rank  order  these  survey  items  according  to  their  individual  level  of  agreement.) 
This  collection  of  items,  termed  the  concourse,  is  not  restricted  to  textual  material  but  has 
been  known  to  include  paintings,  art  work,  photographs,  or  even  musical  selections  [Brown 
93].  While  the  actual  items  may  take  different  forms,  the  scale  on  which  participants  sort  the 
items  remains  fixed.  For  example,  a  participant  may  be  asked  to  rate,  on  a  scale  of  -5  to  +5, 
the  degree  to  which  they  agree  with  the  statement:  "the  software  process  would  benefit  more 
from  additional  investments  in  quality  assurance  (QA)  than  additional  investments  in  system 
design.” 
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The  survey  items  (or  statements)  that  compose  the  concourse  can  be  derived  in  numerous 
ways.  Typically,  the  items  are  collected  through  personal  interviews  and  focus  groups.  Focus 
group  discussions  often  reveal  the  multiplicity  of  subjective  perceptions  and  allow  the 
researcher  to  design  specific  survey  items  that  drill  down  into  particular  attitudes.  Other 
sources  for  survey  items  may  include  journal  publications  or  newspaper  articles.  The  primary 
point  is  that  the  collection  of  items  in  the  concourse  should  reflect  the  range  of  perceptions  on 
a  particular  topic  of  interest.  In  many  cases,  researchers  will  pilot  test  the  concourse  to  verify 
its  validity.  Because  participants  are  asked  to  sort  the  statements  in  a  particular  manner  (see 
Section  2.2),  most  Q-Samples  consist  of  33  items.  However,  they  are  not  restricted  to  33  and 
can  employ  any  number  of  items.  After  the  Q-Sample  is  constructed,  each  item  in  the  sample 
is  numbered  for  data  recording  purposes. 

It  should  be  noted  that  the  goal  of  the  Q-S  ample  is  to  provide,  in  miniature,  a  comprehensive 
portrayal  of  the  larger  process  being  modeled  [Brown  93] .  To  ensure  content  validity,  sample 
statements  are  usually  reviewed  by  domain  experts  and  tested  in  one  or  more  pilot  studies.  In 
terms  of  comprehensiveness  and  representativeness  of  any  given  Q-Sample,  the  design  of  the 
instrument  is  performed  just  as  carefully  as  participant  selection  is  conducted  for  survey 
studies.  Hence,  some  researchers  prefer  to  develop  a  larger  set  of  items  and  then  select  a 
random  sample  of  items  to  administer. 

Because  Q  methodology  does  not  seek  to  make  claims  to  larger  representative  groups,  it  is 
less  concerned  with  participant  sampling  techniques.  There  is  often  the  assumption  that  Q 
methodology  participants  represent  the  total  population.  In  Stage  1 ,  given  the  focus  is  on 
capturing  a  wide  array  of  perceptions,  the  rigor  that  is  often  associated  with  identifying  the 
target  sample  is  redirected  toward  identifying  the  survey  items.  The  target  goal  of  the  Q- 
Sample  is  a  comprehensive,  balanced,  and  representative  set  of  survey  items.  Once  the  Q- 
Sample  has  been  designed  it  is  released  for  “sorting.” 


2.2  Stage  2:  Administering  the  Q-Sort 

Stage  2  involves  administering  the  Q-Sort  or  collecting  participant  perceptions.  When 
administering  the  Q-Sort,  participants  are  often  given  a  sheet  with  specific  sorting 
instructions  called  a  condition  of  instruction  and  an  answer  sheet  to  record  the  rank  ordering 
(see  Figure  1). 


Most  Disagree  Most  Agree 

-4 

-3 

-2 

-1 

+1 

+2 

+3 

+4 

Figure  1:  Q-Sort  Answer  Sheet 
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In  this  stage,  participants  are  asked  to  sort  the  opinion  statements  into  a  predefined  set  of 
categories,  ranging  from  “Most  Agree”  to  “Most  Disagree.”  As  noted  in  Figure  1,  the  answer 
form  that  participants  use  to  sort  the  survey  items  forces  the  Q-Sort  into  the  shape  of  a 
normal  distribution.  There  are  fewer  statements  that  can  be  placed  at  either  end,  allowing 
more  to  be  place  in  the  middle  (or  neutral)  zone  of  the  scale. 

The  Q-Sample  is  administered  to  each  participant  and  his  or  her  individual,  rank-ordered 
statements  are  then  recorded.  The  resulting  data  matrix  will  reflect  the  participant’s  sorting 
arrangements  in  the  column  with  the  survey  item  statement  along  the  row.  The  ranking  level 
of  each  item  is  then  entered  into  the  data  matrix  to  allow  factor  analysis.  Because  each  item  is 
measured  according  to  the  individual’s  perception,  and  on  the  same  scale,  the  individual’s 
specific  array  can  be  correlated  with  the  arrays  of  the  other  participants.  The  data  can  then  be 
analyzed  with  a  variety  of  statistical  tools  (see  Section  2.3).  The  outcomes  of  this  stage 
should  be  a  data  matrix  of  the  participants’  rank  ordering  of  each  of  the  survey  items  in  the 
Q-Sample.  Given  that  respondents,  or  participants,  array  their  perceptions  in  a  forced  matrix 
reflecting  a  normal  distribution  curve,  participants  can  be  correlated  and  grouped  according 
to  their  level  of  agreement  or  similarity — the  goal  of  the  next  stage. 


2.3  Stage  3:  Factor  Analyzing  the  Q-Sort 

Data  analysis  in  Q  methodology  typically  involves  the  sequential  application  of  three  sets  of 
statistical  procedures:  correlation,  factor  analysis,  and  the  computation  of  factor  scores.  The 
discussion  below  speaks  briefly  to  each  of  these  issues.  Readers  are  encouraged  to  examine 
the  citations  provided  below  to  develop  a  deeper  understanding  of  these  specific  statistical 
tools.  It  should  be  noted  that  while  the  steps  are  discussed  below,  most  statistical  software 
generates  these  procedures  automatically.  Hence  the  discussion  is  intended  to  describe  what 
is  occurring,  not  to  provide  a  concrete  list  of  instructions. 

The  first  step  in  analyzing  the  data  is  to  generate  a  correlation  matrix  of  the  participants.  As 
Brown  has  shown  [Brown  93],  it  makes  no  difference  whether  the  coefficients  in  the 
correlation  matrix  are  Pearson’s  r,  Spearman’s  rho,  or  other  commonly  employed 
nonparametric  measures  of  association.  As  a  practical  matter,  the  factoring  process 
commences  once  a  matrix  of  Q-Sort  correlations  is  provided. 

According  to  a  variety  of  research  findings,  it  makes  little  difference  whether  the  specific 
factoring  routine  is  principal  components,  centroid,  or  any  other  available  method. 
Regardless  of  the  precise  procedures  employed,  the  resultant  factor  structures  do  not  appear 
to  differ  from  one  another  in  any  appreciable  respects  [McKeown  &  Thomas  88].  By 
convention,  PCA  with  a  Varimax  rotation  is  the  most  common  routine  employed.  (For 
detailed  elaboration,  see  Adcock  [Adcock  54]  or  Brown  [Brown  80]).  Hence  the  case 
examined  below  focuses  on  analyzing  Q-Sorts  using  PCA  with  the  Varimax  method  of 
orthogonal  rotation. 
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The  result  of  the  factor  analysis  is  factor  loadings.  Factor  loadings  are  in  effect  correlation 
coefficients:  they  indicate  the  extent  to  which  each  Q-Sort  (i.e.,  participant)  is  similar  or 
dissimilar  to  the  composite  factor  array.  In  most  research  applications,  factor  interpretation 
proceeds  on  the  basis  of  factor  loadings.  In  Q  methodology,  on  the  other  hand,  interpretations 
are  based  on  factor  scores  which  reflect  the  extent  of  agreement  among  perceptions  related  to 
the  individual  Q-Sort  statements.  In  Q  methodology  the  presence  of  several  orthogonal 
(independent)  factors  is  evidence  of  different  points  of  view  in  the  participant-sample.  An 
individual’s  positive  loading  on  a  factor  indicates  his  or  her  shared  subjectivity  with  others  on 
that  factor;  negative  loadings,  on  the  other  hand,  are  signs  of  rejection  of  the  factor’s 
perspective  [McKeown  &  Thomas  88].  The  case  presented  in  the  following  section  illustrates 
the  analysis  and  interpretation  of  a  group  of  city  and  county  chief  executive  officers  (CEOs) 
to  understand  the  types  of  information  technology  (IT)  assistance  they  perceived  would  be 
most  helpful  to  local  governments. 
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3  Q  methodology:  An  Application 


The  following  illustration  is  based  on  a  government  agency’s  attempt  to  isolate  potential 
investment  strategies  for  supporting  the  software  efforts  of  13  sister  agencies.  The  study 
sought  to  identify  CEO  perceptions  of  areas  of  consensus  and  conflict  over  how  to  best 
support  their  software  engineering  efforts  in  an  era  of  limited  funds.  The  results  of  the  study 
were  used  to  help  define  the  resource  investment  strategies  for  supporting  the  13 
organizations.  The  study  began  with  a  focus  group  discussion  to  identify  the  relevant 
concourse  of  Q-Sample  items.  Based  on  the  discussion  and  dialogue,  33  items  were  derived, 
and  thus  composed  the  Q-Sample  (see  Table  1). 

Table  1:  Q-Sample  Items 


Item  # 

In  my  opinion  resource  investments  should  be  made  in  assisting  with: 

1 

data  policies 

2 

emerging  privacy  legislation 

3 

emerging  security  legislation 

4 

ethical  use  of  software,  technology,  and  data 

5 

changes  in  business  related  standard  operating  procedures 

6 

changes  in  IT  related  standard  operating  procedures 

7 

ADA  requirements 

8 

data  sharing 

9 

prototyping  new  software  solutions 

10 

technical  requirements  definition 

11 

code  inspections 

12 

test  program  reviews 

13 

screen  design,  format,  and  layout 

14 

disaster  recovery 

15 

engineering  documents 

16 

technical  documentation 

17 

training  investments 

18 

ROI  strategies 

19 

quality  assurance 

20 

risk  management 

21 

estimating  cost  and  schedule 

22 

communications  and  team  building 

23 

visioning/strategic  planning  and  goal  alignment 

24 

contracting  and  outsourcing 

25 

auditing  and  post-mortems 
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26 

productivity  changes 

27 

cost  savings 

28 

process  improvement 

29 

added  value 

30 

leveraging  organizational  information  and  knowledge 

31 

developing  feasible  and  reliable  metrics 

32 

citizen  engagement 

33 

transparency  and  accountability 

The  Q-S  ample  was  administered  and  the  CEOs  were  asked  to  sort  the  items  in  terms  of  their 
level  of  agreement  with  investing  resources  in  the  area  on  a  -4  to  +4  scale.  All  participants 
sorted  their  responses  according  to  a  “forced”  normal  distribution  curve.  Thus  only  a  limited 
number  of  +4s,  +3s,  etc.  were  allowed.  Table  2  illustrates  the  Q-Sort  for  Respondent  2  (R2). 
As  demonstrated  in  the  table,  R2  ranked  item  1  (data  policies)  and  item  28  (process 
improvement)  a  +4.  indicating  a  strong  preference  for  the  greatest  investment.  Conversely, 
items  5  and  6  (changes  in  business  related  standard  operating  procedures  and  changes  in  IT 
related  standard  operating  procedures)  were  rated  a  -4,  indicating  perceptions  of  areas  for 
least  investment. 


Table  2:  Q-Sort  for  Respondent  2 


-4 

-3 

-2 

-1 

+1 

+2 

+3 

+4 

5 

15 

9 

19 

3 

7 

2 

17 

1 

6 

25 

10 

22 

4 

21 

14 

23 

28 

31 

11 

26 

12 

24 

33 

18 

16 

29 

13 

27 

8 

32 

20 

30 

The  data  for  all  13  CEOs  were  collected  and  entered  in  a  data  matrix  (see  Table  3).  In 
keeping  with  the  Q  methodology  process,  individual  respondents  were  listed  along  the 
column  and  Q-S  ample  items  along  the  row. 


Table  3:  Q-Sort  by  Respondent 


Respondent 

Item 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

1 

-2 

4 

3 

-1 

4 

3 

2 

4 

4 

4 

3 

0 

4 

2 

-1 

2 

-3 

0 

-2 

-3 

-3 

-1 

-2 

-1 

2 

-1 

-4 

3 

-2 

0 

0 

-3 

2 

-3 

-3 

-1 

-2 

-1 

1 

4 

-4 

4 

-1 

0 

3 

2 

1 

-4 

0 

-2 

2 

-4 

1 

-2 

-3 

5 

-1 

-4 

0 

3 

4 

2 

4 

2 

3 

3 

-1 

0 

-3 

6 

-1 

-4 

0 

0 

3 

2 

3 

3 

0 

4 

-2 

0 

-3 

7 

0 

1 

-4 

-1 

2 

-1 

-4 

-3 

-1 

-3 

-3 

-1 

-2 

8 
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8 

-1 

2 

4 

4 

3 

2 

0 

0 

2 

-2 

-1 

0 

1 

9 

2 

-2 

2 

2 

2 

4 

3 

0 

1 

1 

-3 

2 

0 

10 

-2 

-2 

-1 

1 

2 

1 

1 

1 

-4 

1 

-1 

1 

-1 

11 

-2 

-2 

-3 

-3 

0 

-3 

-1 

1 

-3 

0 

-3 

-1 

0 

12 

-3 

0 

1 

-2 

1 

2 

-1 

0 

0 

-1 

-2 

-2 

-2 

13 

-3 

0 

-1 

-1 

1 

-2 

-4 

-3 

-4 

-2 

-4 

1 

-2 

14 

-3 

2 

-1 

1 

1 

3 

-3 

-3 

0 

1 

0 

-3 

0 

15 

-4 

-3 

4 

-1 

0 

4 

-2 

-2 

-1 

0 

-2 

-4 

1 

16 

-4 

-2 

-1 

0 

1 

3 

-2 

-2 

-1 

0 

0 

-3 

0 

17 

0 

3 

-2 

-2 

-3 

-2 

2 

1 

2 

-1 

1 

-3 

3 

18 

1 

3 

1 

1 

3 

1 

1 

0 

1 

-1 

3 

-4 

2 

19 

1 

-1 

0 

2 

0 

0 

1 

1 

-1 

1 

2 

2 

3 

20 

1 

0 

0 

-2 

0 

-2 

0 

0 

0 

1 

0 

0 

2 

21 

0 

1 

1 

0 

0 

0 

0 

2 

0 

0 

2 

1 

2 

22 

1 

-1 

2 

3 

0 

-2 

0 

2 

3 

0 

1 

3 

2 

23 

2 

3 

2 

1 

-1 

1 

3 

4 

4 

3 

4 

2 

3 

24 

4 

1 

-2 

0 

-4 

1 

2 

3 

0 

-3 

3 

1 

0 

25 

0 

-3 

1 

-3 

-4 

0 

0 

-4 

-1 

0 

0 

0 

-2 

26 

3 

-1 

0 

1 

-1 

0 

-1 

1 

1 

-2 

4 

3 

1 

27 

3 

1 

2 

-2 

-3 

1 

2 

0 

-3 

-2 

2 

2 

4 

28 

4 

4 

-1 

4 

-1 

0 

4 

3 

1 

2 

1 

1 

-1 

29 

0 

-1 

1 

-4 

-1 

-1 

-1 

-1 

-2 

-3 

0 

4 

1 

30 

2 

1 

-2 

3 

-2 

0 

-1 

-1 

2 

2 

0 

-1 

-1 

31 

1 

-3 

-2 

-4 

-3 

-4 

-2 

-2 

-2 

2 

-1 

-2 

0 

32 

0 

-1 

3 

-1 

-2 

-1 

1 

-4 

3 

-4 

-1 

3 

1 

33 

2 

2 

-3 

2 

-2 

-1 

-2 

-1 

1 

3 

-2 

-1 

-1 

The  next  step  is  to  obtain  the  correlation  matrix  of  the  Q-Sorts  (Table  4).  As  noted  in  the 
table,  R1  correlates  with  R5  in  the  amount  of  .51,  and  a  quick  perusal  down  the  column 
shows  a  weak  correlation  with  R3  and  R4  (.34  and  .32,  respectively).  R3,  on  the  other  hand, 
shows  a  fairly  moderate  correlation  with  R4,  R5,  and  R7  (.69,  .49,  and  .46). 


It  should  be  noted  that  Q-Sort  correlations  are  rarely  of  any  interest  in  and  of  themselves. 
This  is  because  we  are  not  as  interested  in  how  closely  two  people  correlate  as  we  are  in 
tapping  dominant  perceptions  of  the  group  in  its  entirety.  Hence,  the  correlation  matrix 
represents  merely  a  phase  through  which  the  data  pass  on  the  way  to  factor  analysis  [Sell  & 
Brown  84], 
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Table  4:  Correlation  Matrix  between  Sorts 


SORTS 

i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

rl 

100 

29 

34 

32 

51 

23 

27 

23 

14 

1 

13 

31 

-3 

r2 

29 

100 

7 

20 

15 

23 

-49 

-7 

-23 

-9 

26 

38 

-15 

r3 

34 

7 

100 

69 

49 

34 

46 

4 

32 

22 

35 

35 

33 

r4 

32 

20 

69 

100 

37 

47 

35 

25 

46 

24 

8 

22 

34 

r5 

51 

15 

49 

37 

100 

24 

27 

33 

35 

0 

40 

23 

28 

r6 

23 

23 

34 

47 

24 

100 

2 

-4 

-1 

-13 

-11 

32 

6 

r7 

27 

-49 

46 

35 

27 

2 

100 

28 

44 

40 

-12 

-15 

28 

r8 

23 

-7 

4 

25 

33 

-4 

28 

100 

47 

-9 

-6 

-4 

33 

r9 

14 

-23 

32 

46 

35 

-1 

44 

47 

100 

20 

18 

-1 

44 

rlO 

1 

-9 

22 

24 

0 

-13 

40 

-9 

20 

100 

20 

-16 

9 

rll 

13 

26 

35 

8 

40 

-11 

-12 

-6 

18 

20 

100 

38 

37 

rl2 

31 

38 

35 

22 

23 

32 

-15 

-4 

-1 

-16 

38 

100 

23 

rl3 

-3 

-15 

33 

34 

28 

6 

28 

33 

44 

9 

37 

23 

100 

Note:  Decimals  to  two  places  omitted 

Once  the  correlation  matrix  is  obtained,  the  factor  analysis  process  begins  (an  unrotated 
factor  matrix  is  computed  and  Eigenvalues  are  identified).  The  Eigenvalues  reflect  the 
amount  of  variation  accounted  for  by  the  corresponding  factor.  In  essence,  the  relative 
magnitude  of  the  Eigenvalues  can  be  used  to  order  the  importance  of  the  factors.  By 
convention,  factors  with  Eigenvalues  greater  than  1 .00  are  considered  significant;  those  with 
Eigenvalues  of  lesser  value  are  considered  too  weak  to  warrant  serious  attention. 


For  this  size  sample,  factor  loadings  in  excess  of  .45  are  considered  significant  (p<.01). 
Referring  to  Table  5  factor  1,  the  loadings  of  Rl,  R3,  R4,  R5,  R7,  and  R9  (.54,  .79,  .77,  .71, 
.53,  and  .62  respectively)  mean  that  the  Q-Sorts  in  their  totality  for  each  person  demonstrates 
a  common  perspective,  or  are  highly  correlated.  The  Eigenvalue  of  3.73  is  considered 
significant  and  thus,  factor  1  appears  to  represent  a  common  viewpoint  held  by  a  number  of 
the  respondents.  The  Eigenvalues  for  factors  2,  3,  and  4  are  also  significant  (2.3,  1.4,  and  1.3 
respectively).  While  factor  5  appears  to  demonstrate  borderline  significance  (Eigenvalue  = 
1.06),  perusing  the  columns  reveals  that  since  only  two  respondents  (Rl  and  R13)  were 
loaded  significantly,  the  factor  did  not  constitute  a  large  enough  group  to  merit  further 
examination.  Hence  the  first  four  factors  were  identified  as  candidates  for  further  analysis. 


Table  5:  Unrotated  Factor  Loadings 


SORTS 

Factors  1 

2 

3 

4 

5 

6 

7 

8 

rl 

0.5426 

0.2893 

-0.3061 

-0.1161 

0.5690 

-0.1729 

0.2301 

0.0080 

r2 

0.1000 

0.7930 

0.0314 

-0.0216 

0.2049 

0.4741 

0.0322 

0.0536 

r3 

0.7975 

0.0910 

-0.0245 

0.3505 

-0.0821 

-0.1689 

-0.0914 

-0.1712 

r4 

0.7767 

0.0401 

-0.2650 

0.2174 

-0.2016 

0.3568 

-0.0172 

-0.1199 

10 
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r5 

0.7152 

0.1459 

0.0232 

-0.2443 

0.2944  -0.2050 

-0.4195 

0.1424 

r6 

0.3731 

0.4166 

-0.5484 

0.1631 

-0.4020  -0.0046 

-0.1690 

0.2243 

r7 

0.5382 

-0.6394 

-0.2230 

0.2097 

0.1339  -0.2727 

0.1583 

0.0677 

r8 

0.4119 

-0.3223 

-0.1397 

-0.6877 

0.0908  0.2724 

0.1562 

0.1525 

r9 

0.6241 

-0.4545 

0.1129 

-0.2267 

-0.0730  0.2306 

-0.0739 

-0.3979 

rlO 

0.2503 

-0.3609 

0.2595 

0.6549 

0.2832  0.3112 

0.1487 

0.2223 

rll 

0.4066 

0.3035 

0.7821 

0.0344 

0.1238  -0.0660 

-0.1543 

-0.0002 

rl2 

0.3906 

0.6377 

0.1741 

-0.0493 

-0.2238  -0.2336 

0.4577 

-0.1239 

rl3 

0.5619 

-0.2160 

0.4139 

-0.2067 

-0.4724  -0.0300 

0.1149 

0.3137 

Eigenvalues 

3.7395 

2.3121 

1.4293 

1.3139 

1.0617  0.8109 

0.5907 

0.4665 

%  expl.Var. 

29 

18 

11 

10 

8  6 

5 

4 

Table  6:  Factor  Matrix  with  an  X  Indicating  a  Defining  Sort 


Loadings 

Q-SORT 

1 

2 

3 

4 

rl 

-0.1029 

0.6442X 

0.1156 

0.2153 

r2 

-0.3271 

0.3842 

0.5380 

-0.3102 

r3 

0.5105 

0.6165 

0.2706 

0.2320 

r4 

0.3629 

0.7102X 

0.0588 

0.2879 

r5 

0.0005 

0.4702 

0.3222 

0.5178 

r6 

-0.0582 

0.7818X 

-0.0382 

-0.1549 

r7 

0.5853 

0.2367 

-0.3969 

0.4858 

r8 

-0.2864 

0.1088 

-0.1682 

0.8024X 

r9 

0.2711 

0.0995 

0.0066 

0.7595X 

rlO 

0.8270X 

-0.0594 

0.0357 

-0.0223 

rll 

0.2244 

-0.0869 

0.8794X 

0.1973 

rl2 

-0.1428 

0.4107 

0.6344X 

-0.0173 

rl3 

0.2315 

-0.0202 

0.3412 

0.6372X 

%  expl.Var. 

14 

20 

15 

19 
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In  Q  methodology,  interpretations  are  based  on  factor  arrays  and  factor  scores  rather  than  the 
loadings  (as  per  typical  factor  analytic  techniques).  After  controlling  for  the  weighting  of 
each  of  the  factor  loadings,  a  composite  Q-Sort  (termed  the  array )  can  be  uncovered  for  each 
of  the  factors.  As  shown  in  Table  7,  these  factor  arrays  reflect  an  overall  Q-Sample  for  the 
respondents  who  loaded  on  the  factor  in  total.  Based  on  these  factor  arrays,  we  can  identify 
the  distinguishing  statements  (items)  that  are  associated  with  each  factor.  It  is  in  this  way  that 
we  can  begin  to  see  the  pattern  of  thoughts  that  arises  specific  to  each  of  the  four  groups. 

Table  7:  Factor  Q-Sort  Values  for  Each  Statement 


Factor  Arrays 

No. 

Statement 

No. 

1 

2 

3 

4 

1 

data  policies 

1 

-1 

4 

3 

4 

2 

emerging  privacy  legislation 

2 

-2 

-1 

-4 

1 

3 

emerging  security  legislation 

3 

4 

-2 

-1 

-1 

4 

ethical  use  of  software,  technology,  and  data 

4 

-2 

-2 

2 

0 

5 

changes  in  business-related  standard  operation  procedures  (SOP) 

5 

-1 

3 

0 

-4 

6 

changes  in  IT  related  SOP 

6 

-1 

3 

0 

-4 

7 

ADA  requirements 

7 

-2 

-3 

-4 

-1 

8 

data  sharing 

8 

-1 

1 

4 

1 

9 

prototyping  new  software  solutions 

9 

2 

1 

3 

-3 

10 

technical  requirements  definition 

10 

1 

1 

-1 

-2 

11 

code  inspections 

11 

-2 

0 

-4 

-3 

12 

test  program  reviews 

12 

-2 

-2 

2 

-1 

13 

screen  design,  format,  and  layout 

13 

1 

-3 

-2 

-3 

14 

disaster  recovery 

14 

-3 

0 

-1 

1 

15 

engineering  documents 

15 

-4 

-1 

4 

-2 

16 

technical  documentation 

16 

-3 

-1 

-1 

-1 

17 

training  investments 

17 

-3 

-1 

-2 

3 

18 

ROI  strategies 

18 

-4 

0 

1 

3 

19 

quality  assurance 

19 

2 

2 

0 

1 

20 

risk  management 

20 

-1 

1 

-1 

1 

21 

estimating  cost  and  schedule 

21 

1 

1 

1 

2 

22 

communications  and  team  building 

22 

3 

2 

1 

0 

12 
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visioning/strategic  planning  &  goal  alignment 

23 

2 

4 

2 

4 

24 

contracting  and  outsourcing 

24 

1 

0 

-2 

2 

25 

auditing  and  post-mortems 

25 

-1 

-3 

1 

-2 

26 

productivity  changes 

26 

3 

0 

0 

2 

27 

cost  savings 

27 

2 

-2 

2 

3 

28 

process  improvement 

28 

1 

3 

-1 

2 

29 

added  value 

29 

4 

-4 

1 

0 

30 

leveraging  organizational  info  and  knowledge 

30 

-2 

2 

-2 

0 

31 

developing  feasible  and  reliable  metrics 

31 

-2 

-1 

-3 

-2 

32 

citizen  engagement 

32 

3 

-4 

3 

-1 

33 

transparency  and  accountability 

33 

-2 

2 

-3 

0 

Referring  back  to  Table  6,  Respondents  3,  7,  and  10  comprise  factor  1.  As  illustrated  in  Table 
8,  factor  1  identifies  that  the  grouping  shared  strong  opinions  on  the  need  for  investments  in 
“understanding  and  defining  security  legislation”  and  “achieving  added  value”  from  their 
systems.  Note  that  these  two  items  received  a  +4  on  the  Q-Sort.  The  respondents  did  not  see 
“data  policies,”  “disaster  recovery,”  or  “returns-on-investment”  as  areas  for  investment. 


Table  8:  Distinguishing  Statements  for  Factor  I1 


Factors 

1 

2 

3 

4 

No.  Statement 

No. 

RNK SCORE 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

3 

emerging  security  legislation 

3 

4 

1.78* 

-2 

-0.85 

-1 

-0.38 

-1 

-0.29 

29 

added  value 

29 

4 

1.78* 

-4 

-1.51 

1 

0.25 

0 

-0.16 

13 

screen  design,  format,  layout 

13 

1 

0.43 

-3 

-1.20  - 

2 

-0.69 

3 

-1.10 

1 

data  policies 

1 

-1 

-0.03* 

4 

1.68 

3 

1.52 

4 

2.06 

14 

disaster  recovery 

14 

-3 

-1.39 

0 

-0.13 

-1 

-0.15 

1 

0.47 

18 

ROI  strategies 

18 

-4 

-1.84* 

0 

-0.09 

1 

0.47 

3 

1.58 

(P  <  .05  ;  asterisk  (*)  indicates  significance  at  P  <  .01) 

Turning  our  attention  to  Table  9,  Respondents  1,  3,  4,  and  6  illustrate  the  need  for 
investments  in  “standard  operating  procedures  for  IT  and  business  related  practices”  that 

1  Both  the  Factor  Q-Sort  Value  and  the  Normalized  Score  are  shown  for  Tables  8  through  1 1 . 
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impact  system  efforts.  They  also  hang  together  in  their  impressions  of  the  need  for 
investment  in  determining  ways  to  achieve  “transparency  and  accountability  gains”  from 
systems.  They  are  in  agreement,  albeit  not  concerned ,  with  investing  efforts  in  “cost 
savings,”  “added  value,”  and  “citizen  engagement.” 

Table  9:  Distinguishing  Statements  for  Factor  2 


Factors 

1 

2 

3 

4 

No.  Statement 

No. 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

6  changes  in  IT  related  SOP 

6 

-1 

-0.03 

3 

1.64* 

0 

0.16 

-4 

-1.79 

5  changes  in  business  related  SOP 

5 

-1 

-0.03 

3 

1.61* 

0 

0.16 

-4 

-1.58 

33  transparency  and  accountability 

33 

-2 

-0.48 

2 

0.91 

-3 

-1.41 

0 

-0.07 

27  cost  savings 

27 

2 

0.88 

-2 

-0.78* 

2 

0.89 

3 

1.11 

29  added  value 

29 

4 

1.78 

-4 

-1.51* 

1 

0.25 

0 

-0.16 

32  citizen  engagement 

32 

3 

1.33 

-4 

-1.91* 

3 

1.09 

-1 

-0.37 

(P  <  .05  ;  asterisk  (*)  indicates  significance  at  P  <  .01) 

Table  10:  Distinguishing  Statements  for  Factor  3 


Factors 

1 

2 

3 

4 

No.  Statement  No. 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

15  engineering  documents  15 

-4- 

1.84 

-1 

-0.49 

4 

2.05* 

-2 

-1.07 

8  data  sharing  8 

-1 

-0.03 

1 

0.07 

4 

1.83* 

1 

0.38 

4  ethical  use  of  sfw,  technology,  and  data  4 

-2 

-0.93 

-2 

-1.11 

2 

0.76 

0 

-0.18 

12  test  program  reviews  12 

-2 

-0.93 

-2 

-0.52 

2 

0.58 

-1 

-0.68 

2  emerging  privacy  legislation  2 

-2 

-0.48 

-1 

-0.42 

-4 

-1.63 

1 

0.41 

(P  <  .05  ;  asterisk  (*)  indicates  significance  at  P  <  .01) 

14 
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Respondents  2,  11,  and  12  form  a  similar  pattern  of  thinking  on  the  need  for  investment  in 
“engineering  documents,”  “data  sharing,”  “the  ethical  use  of  systems,”  and  “test  program 
reviews”  (Table  10).  “Emerging  privacy  legislation”  is  not  an  area  that  is  deemed  important 
for  investments. 

Table  1 1:  Distinguishing  Statements  for  Factor  4 


Factors 

1 

2 

3 

4 

No. 

Statement 

No. 

RNK SCORE 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

18 

ROI  strategies 

18 

-4 

-1.84 

0 

-0.09 

1 

0.47 

3 

1.58 

17 

training  investments 

17 

-3 

-1.39 

-1 

-0.33 

-2 

-1.10 

3 

1.28* 

32 

citizen  engagement 

32 

3 

1.33 

-4 

-1.91 

3 

1.09 

-1 

-0.37* 

9 

prototyping  new  software  solutions 

9 

2 

0.88 

1 

0.57 

3 

1.21 

-3 

-1.15* 

5 

changes  in  business  related  SOP 

5 

-1 

-0.03 

3 

1.61 

0 

0.16 

-4 

-1.58* 

6 

changes  in  IT  related  SOP 

6 

-1 

-0.03 

3 

1.64 

0 

0.16 

-4 

-1.79* 

(P  <  .05  ;  asterisk  (*)  indicates  significance  at  P  <  .01) 

Finally,  factor  4  is  distinguished  by  the  need  for  investment  in  “training”  and  “return-on- 
investment  strategies”  related  to  IT  systems.  These  respondents  (5,  7,  8,  9,  and  13)  saw  little 
need  for  investments  in  assistance  with  “standard  operating  procedures”  or  “prototyping.” 
Table  12  shows  the  Q-Sample  items  that  did  not  distinguish  any  of  the  factors.  It  should  be 
noted  that  this  does  not  imply  that  none  of  the  respondents  deemed  these  areas  as  necessary 
for  investment;  rather,  their  thinking  on  the  subjects  did  not  distinguish  them  from  the  others. 

Table  12:  Items  That  Do  Not  Distinguish  between  Any  Pair  of  Factors 


Factors 

1 

2 

3 

4 

No.  Statement 

No. 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

RNK  SCORE 

16  technical  documentation 

16 

-3 

■1.39 

-1 

-0.35 

-1 

-0.15 

-1 

-0.52 

19*  quality  assurance 

19 

2 

0.88 

2 

0.76 

0 

-0.05 

1 

0.50 

20*  risk  management 

20 

-1 

-0.03 

1 

0.01 

-1 

-0.27 

1 

0.21 

21*  estimating  cost  and  schedule 

21 

1 

0.43 

1 

0.40 

1 

0.36 

2 

0.87 

22  communications  and  team  building 

22 

3 

1.33 

2 

0.83 

1 

0.56 

0 

0.17 
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23  strategic  planning  and  goal  alignment  23  2 

0.88  4  1.70  2 

0.89 

4 

1.90 

31  developing  feasible  and  reliable  metrics  31  -2 

-0.93  -1  -0.39  -3 

-1.32 

-2 

-0.98 

All  listed  statements  are  non-significant  at  P>.01,  and  those  flagged  with  an  *  are  also  non- 

-significant  at  P>.05. 

In  sum,  the  analysis  revealed  four  patterns  of  thought:  one  that  stressed  security  and  added 
value  (factor  1),  one  that  stressed  process  issues  (factor  2),  one  that  valued  technical  support 
(factor  3),  and  one  that  was  concerned  with  training  and  investment  (factor  4).  Note  that  in 
the  analysis,  we  are  less  concerned  with  which  respondents  fall  where  and  more  concerned 
with  the  overall  pattern  of  opinions  that  emerges.  Q  methodology’s  strength  is  in  revealing 
the  dominant  patterns  and  clusters  of  opinions  that  surface  within  a  group. 
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4  Conclusion 


Q  methodology  is  a  method  for  exploring  dominant  perceptions  or  patterns  of  thought  within 
individuals  and  among  groups.  In  essence,  it  is  a  method  that  can  help  to  capture  and  reflect 
the  richness  and  complexity  of  various  points  of  view.  It  is  used  primarily  to  identify  groups 
with  conflicting  values,  preferences,  and  opinions  to  better  understand  the  differences.  It  is 
also  often  used  to  identify  potential  areas  for  research  or  action.  In  the  practice  of  software 
engineering,  Q  methodology  seems  to  have  potential  application  in  many  areas  including: 

•  Measuring  the  extent  and  nature  of  agreement  on  process  improvement  objectives 

•  Measuring  the  extent  and  nature  of  agreement  on  product  functional  and  quality 
requirements 

•  Supplementing  reviews  of  software  architectures  by  measuring  the  extent  to  which 
reviewers  believe  the  architecture  satisfies  various  quality  requirements 

•  Generally  illuminating  differing  perceptions  and  opinions  among  project  and 
organizational  stakeholders 

Q  methodology  is  a  tool  for  uncovering  perceptions.  It  is  not  a  tool  for  isolating  direct  cause 
and  effect.  In  other  words,  Q  methodology  is  a  tool  for  offering  insight  but  not  for  prediction. 
As  such,  it  complements  and  expands  the  host  of  tools  project  managers  currently  have  at 
their  disposal.  It  fills  a  need  in  that  it  helps  to  isolate  unknowns.  But  what  ultimately  happens 
to  those  revealed  “unknowns”  is  beyond  the  scope  of  Q  methodology. 
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Appendix  A  Glossary 


Concourse 

composite  statement 
arrays 

Q-Sample 

Q-Sort 

condition  of 
instruction 

factor 

factor  loadings 

factor  scores 


A  list  of  items  serving  as  candidates  for  inclusion  in  the  Q-Sort.  It 
can  take  the  form  of  questions,  statements,  pictures,  etc. 

The  composite  response  set  illustrating  the  average  opinions  of  the 
respondents  that  clustered  on  a  factor. 

The  sample  of  items  that  are  drawn  from  the  concourse  and 
comprise  the  instrument  that  will  be  provided  to  the  respondents. 

Each  respondent’s  rank  ordered  set  of  perceptions. 

The  instructions  provided  to  each  respondent. 


The  cluster  of  respondents  whose  Q-Sorts  were  statistically  similar. 

Each  respondent’s  correlation  with  each  of  the  identified  clusters  or 
factors. 

These  scores  show  the  level  of  consensus/conflict  among  statements 
within  each  opinion  cluster.  They  serve  as  the  basis  of 
interpretation. 
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Appendix  B  Relevant  Web  Sites 


Q  Methodology  Network:  http://listserv.kent.edu/archives/q-method.html 
QA  Method  for  Modern  Research:  http://www.qmethod.org 
Q-Sort:  http://www.qmethod.com 

QMethod  Page:  http://www.rz.unibw-muenchen.de/~p41bsmk/qmethod 
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